\chapter{TLMManager System Design}
The TLMManager is responsible for managing the TLM based co-simulation. 
The main component used in the TLMManager is an instance of {\em ManagerCommHandler} that takes a composite model as input.
{\scriptsize
\begin{verbatim}
    //! Constructor.
    ManagerCommHandler(CompositeModel& Model):
                       MessageQueue(),
                       Comm(Model.GetComponentsNum(), 
                            Model.GetSimParams().GetPort()),
                       TheModel(Model),
                       MonitorConnected(false),
                       CommMode(CoSimulationMode),
                       monitorInterfaceMap(),
                       monitorMapLock(),
                       runningMode(StartUpMode),
                       exceptionMsg(""),
                       exceptionLock()
    {
    };
\end{verbatim}
}

A {\em CompositeModel} instance can be generated in different ways. 
The default TLM co-simulation implementation contains an XML composite model reader, see also Figure~\ref{fig:CompositeModelReader}. 
For details about the XML composite model description see Section~\ref{meta}.

\begin{figure}[ht]
\begin{center}
	\begin{tikzpicture}
	\def\dx{0.8cm}
	\node[thick,draw,rectangle,
	      minimum width=3.3*\dx,
	      minimum height=0.5*\dx,
	      font=\sffamily\tiny,
	      text width=3*\dx] () {SimulationParams};
	\node[thick,draw,rectangle,
	      minimum width=3.3*\dx,
	      minimum height=1.8*\dx,
	      font=\sffamily\tiny,
	      yshift=-1.15*\dx,align=left,
	      text width=3*\dx] () {- TimeStart\\- TimeEnd\\- Port\\- MonitorPort\\- Timeout};
	\node[thick,draw,rectangle,
	      minimum width=3.3*\dx,
	      minimum height=4.0*\dx,
	      font=\sffamily\tiny,
	      yshift=-4.05*\dx,align=left,
	      text width=3*\dx] (a) {+ SimulationParams()\\
	      	                     + Set()\\
	      	                     + GetPort()\\
	      	                     + SetPort()\\
	      	                     + GetMonitorPort()\\
	      	                     + SetMonitorPort()\\ 
	      	                     + GetStartTimeStr()\\
	      	                     + SetStartTimeStr()\\
	      	                     + GetEndTimeStr()\\
	      	                     + SetEndTimeStr()\\
	      	                     + GetServerName()\\
	      	                     + GetTimeout()};	
	      
	\node[thick,draw,rectangle,
	      minimum width=5*\dx,
	      minimum height=0.58*\dx,
	      font=\sffamily\tiny,
	      yshift=-7.5*\dx,
	      text width=4.7*\dx] (b) {CompositeModel};
	\node[thick,draw,rectangle,
	      minimum width=5*\dx,
	      minimum height=1.5*\dx,
	      font=\sffamily\tiny,
	      yshift=-8.54*\dx,align=left,
	      fill=white,
	      text width=4.7*\dx] () {- Components\\
	      	                      - ComponentParameters\\
	      	                      - Interfaces\\
	      	                      - Connections\\
	      	                      - SimParams};
	\node[thick,draw,rectangle,
	      minimum width=5*\dx,
	      minimum height=5.5*\dx,
	      font=\sffamily\tiny,
	      yshift=-12.6*\dx,align=left,
	      fill=white,
	      text width=4.7*\dx] (c) {+ CompositeModel()\\
	      	                       + \texttt{\textasciitilde}CompositeModel()\\
	      	                       + RegisterTLMComponentProxy()\\
	      	                       + RegisterComponentParameterProxy()\\
	      	                       + RegisterTLMInterfaceProxy()\\
	      	                       + GetTLMComponentProxy()\\
	      	                       + GetComponentParameterProxy()\\
	      	                       + GetTLMInterfaceProxy()\\
	      	                       + GetComponentsNum()\\
	      	                       + GetComponentParametersNum()\\
	      	                       + GetInterfacesNum()\\ 
	      	                       + GetTLMComponentID()\\
	      	                       + GetComponentParameterID()\\
	      	                       + GetTLMInterfaceID()\\
	      	                       + RegisterTLMConnection()\\
	      	                       + GetTLMConnection()\\
	      	                       + StartComponents()\\
	      	                       + GetSimParams()\\
	      	                       + CheckProxyComm()\\
	      	                       + Print()};	
	      
	\node[thick,draw,rectangle,
	      minimum width=3.3*\dx,
	      minimum height=0.58*\dx,
	      font=\sffamily\tiny,
	      yshift=-17.2*\dx,
	      text width=4.2*\dx,
	      fill=gray!40] (d) {CompositeModelReader};
	\node[thick,draw,rectangle,
	      minimum width=3.3*\dx,
	      minimum height=0.58*\dx,
	      font=\sffamily\tiny,
	      yshift=-17.73*\dx,align=left,
	      text width=4.2*\dx,
	      fill=gray!40] () {- TheModel};
	\node[thick,draw,rectangle,
	      minimum width=3.3*\dx,
	      minimum height=2.8*\dx,
	      font=\sffamily\tiny,
	      yshift=-19.5*\dx,align=left,
	      text width=4.2*\dx,
	      fill=gray!40] () {+ CompositeModelReader()\\
	      	                + ReadModel()\\
	      	                - ReadComponents()\\
	      	                - ReadTLMInterfaceNodes()\\
	      	                - ReadComponentParameters()\\
	      	                - ReadSimParams()\\
	      	                - FindChildByName()\\ 
	      	                - FindAttributeByName()\\
	      	                - ReadTLMConnectionNode()};	
	      	                  
    \draw[thick,->,dashed] (b) --node[right]{\tiny\sffamily SimParams}(a);	      	                  	          	            
    \draw[thick,->,dashed] (d) --node[right]{\tiny\sffamily TheModel}(c);	      	                  	          	            
	\end{tikzpicture}
  %{\includegraphics[width=4cm]{figs/classMetaModelReader__coll__graph.png}}
\end{center}
\caption{UML collaboration diagram of the CompositeModelReader class. The
  CompositeModelReader initialized the CompositeModel data structure.}
\label{fig:CompositeModelReader}
\end{figure}

The {\em CompositeModelReader} parses the XML composite model file and initializes {\em CompositeModel} and {\em SimulationParams} data structures, that is, set-up co-simulation parameters. 
Simulation parameters define global co-simulation settings, e.g., start and end time. 
Composite model parameters define the co-simulation components (simulation models) and their interconnection.

The {\em ManagerCommHandler} is then started, that is, the {\em Run()} function is invoked. 
This function initialized the different threads that are used during the co-simulation:
{\scriptsize
\begin{verbatim}
// Run method executes all the protocols in the right order:
// Startup, Check then Simulate
void ManagerCommHandler::Run(CommunicationMode CommMode_In) {
    CommMode = CommMode_In;

    pthread_attr_t attr;
    pthread_attr_init(&attr);
    pthread_attr_setscope(&attr,  PTHREAD_SCOPE_SYSTEM);
    pthread_t reader, writer;

    // Start the minitoring thread
    pthread_t monitor;
    if(CommMode == CoSimulationMode) {
        pthread_create(&monitor, &attr, thread_MonitorThreadRun, (void*)this);
    }

    // start the reader & writer threads
    pthread_create(&reader, &attr, thread_ReaderThreadRun, (void*)this);

    pthread_create(&writer, &attr, thread_WriterThreadRun, (void*)this);

    // Wait until all threads are finished.
    if(CommMode == CoSimulationMode) {
        pthread_join(monitor, NULL);
    }
    
    pthread_join(reader, NULL);
    pthread_join(writer, NULL);
}
\end{verbatim}
}

Note, that all communication between the co-simulation participants (simulators with models) is going through the TLM manager. 
This communication is handled by the threads that are started in the {\em Run()} function. 
The following threads are started:
\begin{description}
\item[The reader thread] initialized the co-simulation, processes incoming messages, and creates messages to be sent. 
This threads first task is to run the {\em RunStartupProtocol()} that starts all the simulation tools that are participating in the co-simulation and runs the initiation protocol. 
After that the thread goes into message passing mode. 
Messages from one co-simulation participant to another are taken by the reader thread and marshaled to the receiver, that is, they are converted into outgoing messages.
\item[The writer thread] processes all outgoing messages. 
It loops through the queue of outgoing messages and distributes them to the correct receivers.
\item[The monitor thread] copies and forwards all outgoing messages to any connected monitoring process. 
This can be used to monitor the co-simulation. 
If no monitoring process is connected no copying and forwarding will take place.
\end{description}

\begin{figure}
    \begin{center}
    	\begin{tikzpicture}[font=\sffamily\tiny]
    	\node[rectangle,draw,thick,align=left,
    	      minimum width=4cm,
    	      fill=gray!40, 
    	      text width=3.7cm] (managercommhandler) {\hspace{0.7cm} ManagerCommHandler\\
    	      	
    	      	                                      - MessageQueue\\
    	      	                                      - Comm\\
    	      	                                      - TheModel\\
    	      	                                      - CommMode\\
    	      	                                      - monitorInterfaceMap\\
    	      	                                      - monitorMapLock\\
    	      	                                      - runningMode\\
    	      	                                      
    	      	                                      + ManagerCommHanler()\\
    	      	                                      + Run()\\
    	      	                                      + RunStartupProtocol()\\
    	      	                                      + ProcessRegComponentMessage()\\
    	      	                                      + ProcessRegParameterMessage()\\
    	      	                                      + ProcessRegInterfaceMessage()\\
    	      	                                      + ReaderThreadRun()\\
    	      	                                      + WriterThreadRun()\\
    	      	                                      + MarschalMessage()\\
    	      	                                      + MonitorThreadRun()\\
    	      	                                      + getRunState()\\
    	      	                                      + thread\_ReaderRhreadRun()\\
    	      	                                      + thread\_WriterThreadRun()\\
    	      	                                      + thread\_MonitorThreadRun()\\
    	      	                                      - SetupInterfaeConnectionMessage()\\
    	      	                                      - SetupInterfaceRequestMessage()\\
    	      	                                      - UnpackAndStoreTimeData()\\
    	      	                                      - ProcessInterfaceMonitoringMessage()
    	      	                                      - ForwardToMonitor()\\
    	      	                                      - Terminate()};
    	\draw[thick] (-2cm,3.35cm) -- (2cm,3.35cm);
    	\draw[thick] (-2cm,1.35cm) -- (2cm,1.35cm);
    	
    	
    	
    	\node[rectangle,draw,thick,align=left,
    	minimum width=3.6cm,
    	yshift=8cm, 
    	text width=3.3cm] (tlmmanagercomm) {\hspace{0.6cm} TLMManagerComm\\
    		
    		- CurFDSet\\
    		- ContactSockets\\
    		- ClientSockets\\
    		- ActiveSockets\\
    		- StartupMode\\
    		- ServerPort\\
    		- NumClients\\
    		
    		+ TLMManagerComm()\\
    		+ CreateServerSocket()\\
    		+ SelectReadSocket()\\
    		+ HasData()\\
    		+ ClearActiveSockets()\\
    		+ AddActiveSocket()\\
    		+ DropActiveSocket()\\
    		+ SwichToRunningMode()\\
    		+ AcceptComponentConnections()\\
    		+ closeAll()\\
    		+ GetServerPort()};
    	
    	\draw[thick] (-1.8cm,10.25cm) -- (1.8cm,10.25cm);
    	\draw[thick] (-1.8cm,8.25cm) -- (1.8cm,8.25cm);
    	
    	\node[rectangle,draw,thick,align=left,
    	minimum width=2.7cm,
    	yshift=8cm, 
    	xshift=3.6cm,
    	text width=2.4cm] (tlmmessagequeue) {
    		    \hspace{0.0cm} TLMMessageQueue\\
    	   		
    	   		- SendBufLock\\
    	   		- SendBuffers\\
    	   		- FreeBufLock\\
    	   		- FreeBuffers\\
    	   		- SenderWait\\
    	   		- Terminated\\
    	   		
    	   		+ TLMMessageQueue()\\
    	   		+ \textasciitilde TLMMessageQueue()\\
    	   		+ GetReadSlot()\\
    	   		+ PutWriteSlot()\\
    	   		+ GetWriteSlot()\\
    	   		+ ReleaseSlot()\\
    	   		+ Terminate()};
    	\draw[thick] (2.25cm,9.63cm) -- (4.95cm,9.63cm);
 	    \draw[thick] (2.25cm,7.9cm) -- (4.95cm,7.9cm);

    	\node[rectangle,draw,thick,align=left,
    	minimum width=4.2cm,
    	yshift=8cm, 
    	xshift=-4.4cm,
    	text width=3.9cm] (compositemodel) {
    		\hspace{0.0cm} CompositeModel\\
    		
    		- Components\\
    		+ ComponentParameters\\
    		- Interfaces\\
    		- Connections\\
    		- SimParams\\
    		
    		+ CompositeModel()\\
    		+ \texttt{\textasciitilde}CompositeModel()\\
    		+ RegisterTLMComponentProxy()\\
    		+ RegisterComponentParameterProxy()\\
    		+ RegisterTLMInterfaceProxy()\\
    		+ GetTLMComponentProxy()\\
    		+ GetComponentParameterProxy()\\
    		+ GetTLMInterfaceProxy()\\
    		+ GetComponentsNum()\\
    		+ GetComponentParametersNum()\\
    		+ GetInterfacesNum()\\ 
    		+ GetTLMComponentID()\\
    		+ GetComponentParameterID()\\
    		+ GetTLMInterfaceID()\\
    		+ RegisterTLMConnection()\\
    		+ GetTLMConnection()\\
    		+ StartComponents()\\
    		+ GetSimParams()\\
    		+ CheckProxyComm()\\
    		+ Print()};
    	\draw[thick] (-2.3cm,11.1cm) -- (-6.5cm,11.1cm);
    	\draw[thick] (-2.3cm,9.65cm) -- (-6.5cm,9.65cm);

        \node[rectangle,draw,thick,align=left,
        minimum width=2.5cm,
        yshift=15cm, 
        xshift=-4.4cm,
        text width=2.2cm] (simulationparams) {
        	\hspace{0.0cm} SimulationParams\\
        	
        	- TimeStart\\
        	- TimeEnd\\
        	- Port\\
        	- MonitorPort\\
        	- TimeOut\\
        	
        	+ SimulationParams()\\
        	+ Set()\\
        	+ GetPort()\\
        	+ SetPort()\\
        	+ GetMonitorPort()\\
        	+ SetMonitorPort()\\    		
        	+ SetStartTimeStr()\\
        	+ SetEndTimeStr()\\
        	+ GetServerName()\\
        	+ GetTimeout()};
        \draw[thick] (-3.15cm,16.9cm) -- (-5.65cm,16.9cm);
        \draw[thick] (-3.15cm,15.4cm) -- (-5.65cm,15.4cm);

 	    \draw[thick,->,dashed] (managercommhandler) --node[right]{TheModel} (compositemodel); 	    
 	    \draw[thick,->,dashed] (managercommhandler) --node[right]{Comm} (tlmmanagercomm);
   	    \draw[thick,->,dashed] (managercommhandler) --node[right]{MessageQueue} (tlmmessagequeue);
   	    \draw[thick,->,dashed] (compositemodel) --node[right]{SimParams} (simulationparams);
    	\end{tikzpicture}
%    \includegraphics[width=9cm]{figs/classManagerCommHandler__coll__graph.png}
      \caption{ManagerCommHandler collaboration and class diagram.}
      \label{fig:ManagerCommHandlerClassDesign}
    \end{center}
\end{figure}

The {\em ManagerCommHandler} collaborates with different classes to handle co-simulation start-up and communication, see also Figure~\ref{fig:ManagerCommHandlerClassDesign}. 
{\em TLMManagerComm} is responsible for all socket communications on the TLMManager side. The {\em TLMMessageQueue} handles the queue for outgoing messages.

\subsection{Co-Simulation Start-up}
During co-simulation start-up the TLMManager starts the different co-simulation components (external models) and then waits until all components have registered themselves. 
This is done with a simple initialization protocol.

First the components are started using {\tt TLMComponentProxy::StartComponent(...)}. 
This is done using the OS specific execution method, for instance, with {\it fork()} and {\it execlp(...)} on Linux:
{\scriptsize
\begin{verbatim}
  execlp( StartCommand.c_str(), StartCommand.c_str(),
          Name.c_str(),
          startTime.c_str(),
          endTime.c_str(),
          strMaxStep.c_str(),
          serverName.c_str(),
          ModelName.c_str(),
          NULL );
\end{verbatim}
}

The start command is the start-up script that must be provided for each specific simulation tool. 
It takes a couple of parameters as specified in the composite model XML file:
\begin{description}
\item[Name] of the component as specified in the composite model XML file.
\item[Start time] of the simulation in seconds, typically 0.
\item[End time] of the simulation in seconds.
\item[Max step] for the solver to take. This is a limit set by the TLM interfaces of the specific component. 
The solver is not allowed to take larger steps.
\item[Server name] is typically the hostname or IP address that the TLMManager is running on and the port number that it is using. 
For  instance {\it 163.157.1.23:1111}.
\item[Model name] of the simulation model. Typically the input file
  needed to start the external model simulation.
\end{description}

\subsection{External Model Initialization}
After external model start-up, the TLMManager waits until all external models have registered themselves. 
This is performed in {\em ManagerCommHandler::RunStartupProtocol()}. 
Here an outline of the code:
{\scriptsize
\begin{verbatim}
void ManagerCommHandler::RunStartupProtocol() {
    ...
    // Start the external components forming "coupled simulation"
    TheModel.StartComponents();

    // Setup timer
    tTM_Info tInfo;
    TM_Init(&tInfo);
    TM_Start(&tInfo);

    while( (numToRegister > 0) ||
           ( numCheckModel < TheModel.GetComponentsNum()) ) {
        ...

        // Check all the registered components if they send
        // interface registration messages.
        for(int iSock =  TheModel.GetComponentsNum() - 1; iSock >= 0 ; --iSock) {
            ...
            TLMMessage* message = MessageQueue.GetReadSlot();
            message->SocketHandle = hdl;
            TLMCommUtil::ReceiveMessage(*message);

            if(message->Header.MessageType == TLMMessageTypeConst::TLM_CHECK_MODEL) {
                // This component is done with registration. It's will wait for others
                TLMErrorLog::Log(string("Component ") + comp.GetName() + " is ready to simulation");;

                comp.SetReadyToSim();
                numCheckModel++;
            }
            else if(message->Header.MessageType == TLMMessageTypeConst::TLM_REG_PARAMETER) {
                TLMErrorLog::Log(string("Component ") + comp.GetName() + " registers parameter");
                      
                Comm.AddActiveSocket(hdl);
                ProcessRegParameterMessage(iSock, *message);
                MessageQueue.PutWriteSlot(message);
            }
            else {
                TLMErrorLog::Log(string("Component ") + comp.GetName() + " registers interface");;

                Comm.AddActiveSocket(hdl); // expect more messages
                ProcessRegInterfaceMessage(iSock, *message);
                MessageQueue.PutWriteSlot(message);
            }
        }

        // Check if a new connection is waiting to be accepted.
        if((numToRegister > 0) && Comm.HasData(acceptSocket)) {
            int hdl = Comm.AcceptComponentConnections();
            TLMMessage* message = MessageQueue.GetReadSlot();
            message->SocketHandle = hdl;

            if( !TLMCommUtil::ReceiveMessage(*message) ){
                TLMErrorLog::FatalError("Failed to get message, exiting");
                abort();
            }

            ProcessRegComponentMessage(*message);

            MessageQueue.PutWriteSlot(message);
            numToRegister --;
            if(numToRegister == 0)
                TLMErrorLog::Log("All expected components are registered");

            Comm.AddActiveSocket(hdl);
        }

        if(numToRegister)  // still more connections expected
            Comm.AddActiveSocket(acceptSocket);

    }
}
\end{verbatim}
}
\noindent The above method has three main tasks.
First, it checks for new connections from any external model. 
Second, it checks if already registered components need to register any parameters.
Finally, it checks if already registered components need to register any TLM interfaces.

The initialization protocol looks the following way:
\begin{enumerate}
\item An external model send a TLMMessageTypeConst::TLM\_REG\_COMPONENT message to the TLMManager.
\item The TLMManager answers with the same message but also sets the data size of the message to 0.
\item The external model send an interface registration message (TLMMessageTypeConst::TLM\_REG\_INTERFACE).
\item The manager answers with the same message head and adds the connection parameters as stored in the composite model XML file to the data section of the message.
\item The external model repeats the interface registration for all its external (TLM) interfaces.
\item The external model send a parameter registration message (TLMMessageTypeConst::TLM\_REG\_PARAMETER).
\item The manager answers with the same message head and adds the default parameter value as stored in the composite model XML file to the data section of the message.
\item The external model repeats the parameter registration for all its model parameters.
\item Finally the external model sends a TLMMessageTypeConst::TLM\_CHECK\_MODEL that tells that it is ready to simulate.
\item The TLMManager puts the component into {\em ready for simulation} mode.
\end{enumerate}

